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ABSTRACT 

Introduction: Governments in different countries have 
committed to better use of evidence from research in 
policy. Although many programmes are directed at 
assisting agencies to better use research, there have 
been few tests of the effectiveness of such 
programmes. This paper describes the protocol for 
SPIRIT (Supporting Policy In health with Research: an 
Intervention Trial), a trial designed to test the 
effectiveness of a multifaceted programme to build 
organisational capacity for the use of research evidence 
in policy and programme development. The primary 
aim is to determine whether SPIRIT results in an 
increase in the extent to which research and research 
expertise is sought, appraised, generated and used in 
the development of specific policy products produced 
by health policy agencies. 

Methods and analysis: A stepped wedge cluster 
randomised trial involving six health policy agencies 
located in Sydney, Australia. Policy agencies are the 
unit of randomisation and intervention. Agencies were 
randomly allocated to one of three start dates (steps) 
to receive the 1-year intervention programme, 
underpinned by an action framework. The SPIRIT 
intervention is tailored to suit the interests and needs 
of each agency and includes audit, feedback and goal 
setting; a leadership programme; staff training; the 
opportunity to test systems to assist in the use of 
research in policies; and exchange with researchers. 
Outcome measures will be collected at each agency 
every 6 months for 30 months (starting at the 
beginning of step 1). 

Ethics and dissemination: Ethics approval was 
granted by the University of Western Sydney Human 
Research and Ethics Committee HREC Approval H8855. 
The findings of this study will be disseminated broadly 
through peer-reviewed publications and presentations 
at conferences and used to inform future strategies. 
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INTRODUCTION 

Internationally, there is a growing commit- 
ment to better use of evidence from research 
in policy, with a view to improving outcomes 
and optimising resource allocation/"^ As a 
result, considerable attention has recently 



been given to assisting government agencies 
to better use evidence from research in 
policy and programme design. 

Health policies and programmes have been 
a particular focus, and a number of 
approaches to building skills and organisa- 
tional capacity to use evidence from research 
have been developed.^ ^ The target for these 
programmes is government agencies who 
develop national or regional policy and pro- 
grammes rather than organisations that 
provide health services and clinical care. 
Despite the proliferation of such programmes 
and their cost to government, there have 
been few examinations of their impact on the 
use of research. A recent comprehensive 
review^ found only five studies examining the 
impact of strategies to increase the use of evi- 
dence in policy and since that time, to the 
best of our knowledge, only one other rele- 
vant study protocol has been published.^ 

The paucity of trials may be partly a result 
of the considerable challenges in evaluating 
the impact of strategies designed to increase 
the use of research in policy and programme 
design. Agencies involved in policy and pro- 
gramme development, even within health, 
are heterogeneous — they will be different in 
remit, size, proximity to central government 
and in their need for and current capacity to 
use evidence from research. These agencies 
are also often complex organisational systems 
and may respond very differently to the same 
strategy programme. Effective strategies will 
therefore most likely be multicomponent 
and highly tailored to meet the needs of dif- 
ferent agencies. Study designs will need to be 
capable of assessing complex interventions 
and of carefully measuring the context in 
which the programme is delivered; it is most 
likely that the programme will need to be 
delivered at the level of the health policy 
agency, which means a cluster design will be 
required. It may not be feasible to recruit 
and intervene with sufficient agencies to 
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power a randomised controlled trial. A further challenge 
lies in the lack of established validated measures of the 
use of research in policy and programme development; 
in a trial context, an objective measure of the impact of 
the programme is required. 

This paper describes SPIRIT (Supporting Policy In 
health with Research: an Intervention Trial), an evaluation 
of the impact of a multifaceted programme designed to 
build the capacity of health policy agencies' to use research 
in policy and programme development. The trial is based 
on the SPIRIT Action Framework described in detail else- 
where; the Framework hypothesises that the use of research 
in policy is mediated by the capacity of the organisation to 
use research (including the extent to which the agency 
values research, has systems in place to use research and 
the skills of its staff). It predicts that greater capacity will 
lead to more research engagement actions (demonstrated by 
accessing and appraising existing research, generating new 
research and interacting with researchers) and that, in 
turn, more engagement actions will result in a greater use of 
research evidence. 

SPIRIT is a stepped wedge trial evaluating a complex 
intervention and using an objective measure for its 
primary outcome. It extends previous studies which 
often evaluated only one strategy, did not use object- 
ive outcome measures^^ and were unable to 
include a detailed process evaluation. In addition, 
SPIRIT is the only trial designed to bring about organ- 
isational change in policy and programme agencies and 
that has the capacity to assess the long-term impact of 
the programme (up to 18 months after its completion). 

AIMS 

Objective and aims 

The objective of the trial is to examine what impact the 
SPIRIT intervention has and how it works in the field. 
Our primary outcome measure is an objective assess- 
ment made by an expert panel, our secondary measures 
utilise self-informant or key informant reports and our 
process evaluation includes coded and descriptive 
accounts of the intervention delivery and participation 
made during observations of sessions, semistructured 
interviews and written participant feedback. We will 
examine these data across the components of the 
SPIRIT Action Framework (capacity, research engage- 
ment actions and use of research evidence); this will 
enable us to examine the impact of the SPIRIT interven- 
tion programme and to test the causal relationships 
hypothesised by the model. By capacity, we mean the 
extent to which the agency values research, has systems 
in place to use research and the skills of its staff. By 



^For the purposes of SPIRIT, a health policy agency is defined as: A 
body within a state or federal government department, or a statutory 
authority, whose focus is to develop policy which has an impact on 
state-wide or national services and programmes intended to improve 
individual, family or community health. 



research engagement actions, we mean the extent to which 
research is accessed, appraised and generated, and 
where interaction with researchers occurs, in relation to 
the development of a specific policy or programme 
document. By research use, we mean the extent to which 
research is sought and used in developing the policy or 
programme document, taking into account barriers and 
facilitators. 

The specific aims are: 
Primary outcome 
1. To determine, using an objective measure, whether 
the SPIRIT intervention results in an increase in the 
extent to which research engagement actions are 
undertaken, and research is used in the development 
of policy and programme documents. 
Secondary outcome 
1. To determine, using the self-report of staff members, 
whether the SPIRIT intervention results in an 
increase in: 

► The capacity of health policy agencies to use 
research in terms of: (A) the value placed on the 
use of research evidence by individual policy- 
makers and by the organisation; (B) the confi- 
dence of policymakers in undertaking research 
actions and using research and (C) the systems 
and tools the organisation has in place to support 
research use; 

► Research engagement actions; 

► The use of research. 

Process evaluation: The aim of the process evaluation is 
to complement the outcome measures by exploring how 
and why the intervention worked or did not work in dif- 
ferent contexts. This includes descriptions of: 

1. The implementation of the intervention programme, 
including the delivery of the essential elements ( fidelity) ; 

2. Participation in and responses to the intervention 
(eg, how people interacted with the programme, how 
they evaluated different aspects of it and what sort of 
change they experienced, if any) ; 

3. Contextual factors that might have affected the pro- 
gramme and responses to it (eg, agency priorities, 
practice norms, other training or organisational 
change initiatives, relationships with external bodies, 
legislative reform). 

Cost analysis 

1. To document the cost of delivering the intervention. 

IVIetliods 

SPIRIT started in October 2012, and at the time of 
writing, approximately one-half of the total intervention 
period of 30 months has been completed. 

Trial design 

SPIRIT is a stepped wedge cluster intervention trial 
(figure 2) involving six agencies. Two agencies are ran- 
domly assigned to each step. This design is a variation on a 
cross-over design where each intervention unit is measured 
in the control and intervention phases, except that the step 
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Figure 1 The programme logic 
for Supporting Policy In health 
with Research: an Intervention 
Trial (SPIRIT). 



PROGRAMME DESIGN 
KEY CAPACITIES 



EXAMPLE INTERVENTION STRATEGIES 



Organisation and staff 
value research 

Organisation has tools 
and systems to support 
the use of research 
engagement actions 

Staff have the knowledge 
to use and generate 
research 



Leaders encouraged to emphasise value of 
research 

Policy experts demonstrate the practical 
research value 

Engagement with relevant researchers facilitated 

Staff trained to research appraisal 

Staff taught to commission research effectively 



IMPACT OF THE PROGRAMME 



INCREASED CAPACITY 



Organisation and staff value 
research more 

More tools and systems are in 

place to support research 
engagement actions and use 
of research 

Staff have greater knowledge 

and skills in research 
engagement actions and use 
of research 



INCREASED RESEARCH 
ENGAGEMENT ACTIONS 



Research more frequently 
accessed 

Research more frequently 
appraised 

Research more frequently 
commissioned 

More interaction with 
researchers 




More use and more 
appropriate use of research 
findings in policy products 



wedge design involves a cross-over in one direction only. 
The key feature of this design is that the intervention is 
rolled out sequentially over a number of time periods, 
such that all trial units have received the intervention by 
the end of the trial, although the order in which they 
receive it is determined at random/^ Outcome mea- 
sures are obtained at the same time in all sites, at base- 
line and after implementation in each site. This design 



was selected in accordance with the UK Medical 
Research Council^ ^ recommendations because the inter- 
vention is delivered at the level of the health policy 
agency (cluster) and recruiting and intervening with a 
large enough number of policy agencies to power a trad- 
itional randomised controlled trial was unfeasible due 
to the resource-intensive nature of SPIRIT and its status 
as a new, untested programme. 
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Participants 

Recruitment of health policy agencies 

The inclusion and exclusion criteria for policy agencies 
were established prior to recruiting trial participants. An 
agency was eligible to participate in SPIRIT if: (A) a sig- 
nificant proportion of its work was in health policy or 
programme development; (B) at least 20 staff members 
were involved in policy or programme design, develop- 
ment or evaluation; and (C) it was located in Sydney 
(for ease of provision of the programme). 

Government websites which list all New South Wales 
and Australian government health policy and pro- 
gramme agencies were used to identif)^ potentially eli- 
gible agencies located in Sydney, New South Wales 
(where most of the major health policy agencies in the 
state are located). Information from the website of each 
agency was reviewed by members of the investigator 
team with policy experience to exclude those agencies 
without a significant focus on health and on policy or 
programme design, development or evaluation. 
Seventy-five agencies were classified as potentially eli- 
gible and email and/or phone contact was made with 
each agency to determine the number of relevant staff 
members. Following this step, the 16 agencies still 
regarded as potentially eligible were ranked to deter- 
mine those with the greatest specific focus on health 
and the largest numbers of relevant staff. A visit was 
made by two of the authors (SR and NL) to the top six 
ranked agencies, all of whom agreed to take part. The 
agencies which agreed to participate in SPIRIT comprise 
a major centre within the New South Wales Ministry of 
Health and five agencies (four state and one national) 
that develop policy and programmes about specific 
aspects of health. 

Intervention 

Development of the intervention programme 

The intervention programme is underpinned by the 
programme logic shown in figure 1. 

The intervention was designed to be appropriate for 
policy agencies, taking into account that they are sophis- 
ticated, complex organisations with skilled staff and 
diverse policy priorities. The design of the programme 
was strongly influenced by cognitive behavioural 
theory,^^ system science,^^ organisational change 
theory and adult learning theories. We used 
these approaches, together with information from inter- 
views with policymakers and experience from existing 
programmes in Australia^^"^^ and internationally,^ ^ 
to identify factors that have been associated with signifi- 
cant individual or organisational change. Telephone 
interviews were then conducted with nine senior 
Australian health policymakers to canvass their views 
about the kinds of strategies being considered for inclu- 
sion in the intervention. They endorsed our approach, 
indicating that the proposed change strategies were 
acceptable and would most likely make a difference in 
improving the use of research evidence in policymaking. 



Components of the intervention and all three mea- 
sures were piloted in a policy agency that was not part of 
SPIRIT prior to the start of the study. This pilot broadly 
confirmed that the proposed methods were acceptable 
and appropriate and allowed fine tuning of some 
aspects of the intervention. 

SPIRIT: THE PROGRAMME 

The SPIRIT intervention is a six-component programme 
(table 1): 

1 . Audit, feedback and goal setting: SPIRIT begins with the 
provision of a facilitated discussion of the findings 
from the agency's preintervention measures (see 
below) to the individuals that each agency nominates 
to be part of their 'leaders' group. One of the authors 
(SR) facilitates these sessions and encourages the 
leaders to reflect on and discuss the current strengths 
of the agency in using evidence from research, and 
opportunities for improvement based on the preinter- 
vention data. The discussion is used to (A) agree on 
priorities for change; (B) select the elective elements 
of the SPIRIT programme (see table 1) and (C) iden- 
tify other actions outside the scope of the SPIRIT pro- 
gramme that the agency might wish to undertake to 
increase its use of evidence in policy. 

2. Leadership programme: This component provides an 
opportunity for agency-nominated leaders to con- 
sider the value of research in their work, current 
approaches to knowledge exchange and how to bring 
about a greater use of evidence from research in 
their agency. It draws on the CFHCI EXTRA pro- 
gramme,^ and consists of two interactive workshops 
led by experts in knowledge exchange and policy 
development. 

3. Agency support for research: This component is designed 
to strengthen the perception among staff that their 
organisation values the use of evidence and to 
provide tools for structural change. It includes (A) a 
quarterly email sent to all staff in each agency by the 
chief executive (CE). The email aims to demonstrate 
CE support for the use of evidence in the agency's 
work and is drafted by the research team and contex- 
tualised by the CE; (B) access to Web CIPHER 
(Centre for Informing Policy in Health with Evidence 
from Research), an online resource for policymakers 
where users can access research, hear from opinion 
leaders, share knowledge and debate ideas about the 
use of research in policymaking and programme 
development and (c) provision of relevant fact sheets 
and key publications. 

4. Systems for accessing research, commissioning a review and 
analysing relevant data: This component gives agencies 
an opportunity to trial existing systems for helping 
agencies to use research evidence. The agency elects 
to trial the system for one of: a brokered rapid review 
of research evidence around a topic of interest; a bro- 
kered analysis of local data or the development of an 
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Table 1 Overview of the intervention components and subcomponents, their delivery mode and goals 



Components 



Intervention subcomponents/ 
sessions 



Format/mode 



Goals 



Audit, feedback and 
goal setting 



Leadership programme 




Agency support for 
research 



Opportunity to test 
systems for accessing 
research and reviews 



Research exchange 



1 . Feedback forum* 



1 . Leading organisational change* 

2. Supporting the use of evidence in 
the team* 



1 . Quarterly email endorsements 
from the participating agency 
CEO* 

2. Web CIPHER 

3. Tools and resources 



1 . A brokered rapid review or the 
development of an evaluation plan 
or a brokered analysis of local 
linked dataf 



1 . Research exchanges A 

2. E-bulletinsA 



} 



Interactive feedback (from the preintervention 
measures) forum for executive staff. Opportunities 
for improvement are identified. 
Further feedback sessions are provided 
mid-intervention and at the conclusion of the trial 
Interactive forums for executive staff. Forum 1 
presents feedback about the agency's use (see 
above). Forums 2 and 3 explore how leaders can 
build capacity. Forums provided by policy experts 
and leaders in knowledge exchange. Real world 
examples and collaborative problem-solving used 

1 . CEO email to all staff supporting SPIRIT, 
promoting the use of research in agency and 
providing links to relevant resources 

2. Access via Web CIPHER to new research, 
evidence libraries and research expertise 

3. Provision of useful information at R4P 
sessions, such as key publications from 
leaders in knowledge exchange 

Intervention officers and nominated staff negotiate 
the product requirements (research questions, 
timelines, deliverables, etc). Expert researchers 
develop the product or consult on its development. 
Agency staff may be as involved in the process of 
developing the product as they wish 

1 . A tailored interactive form that brings together 
researchers and policymakers around a topic 
specified by the agency 

2. Receipt of a summary of recently published 
systematic reviews relevant to the agency's 
work 



Leaders make informed decisions about 
opportunities for growing their agencies' capacity 
to use research 



Leaders can identify modifiable barriers to 
research-informed policy/programme development 
and identify strategies for reducing them in their 
agencies 



1 . Staff see that their senior leader values the use 
of research in policy/programme work and that 
they (staff) have increased regard for the value 
of research in policy/programme work 

2. Staff are given access to research resources 

3. Staff attending R4P events receive hardcopy 
resources to guide action responses 



Staff gain experience commissioning a review/an 
evaluation/a piece of linked data analysis 
Agencies have a research product that directly 
addresses a current priority or concern and use it 
to inform their policy/programme work 

Staff receive accessible research information that 
directly addresses current priorities or concerns 
and use it to inform their policy/programme work 
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evaluation framework related to an aspect of the 
agency's work. A tailored research product is pro- 
duced for each agency as a result of this process. 

5. Research access: In this component, agencies choose 
three occasions of facilitated access to research or 
researchers — this might be (A) a tailored interactive 
forum that brings together researchers and policy- 
makers around a topic specified by the agency; or 
(B) receipt of a summary of recently published sys- 
tematic reviews relevant to the agency's work. 

6. Staff training and skill development This component is 
designed to increase the value placed on the use of 
research in policy (symposium 1, received by all agen- 
cies) and to increase skills in areas selected by the 
agency (symposia 2-3, topics selected by the agency) . 
The workshops are provided by individuals with 
expertise appropriate to the topic, include tailored 
case studies and draw on adult learning principles. 
Symposium 1 includes feedback on the current use 
of evidence from research by the agency and sympo- 
sia 2 and 3 might address any of the following 
depending on the interests of the agency: accessing 
and applying systematic reviews; skills for appraising 
research; policy and programme evaluation (intro- 
duction to evaluation or evaluation in practice); 
working with researchers or commissioning research 
for policy and programme development. 

The active phase of the intervention runs for 
12 months with around one activity in each agency per 
month. There are no costs to agencies for any compo- 
nent, other than the opportunity cost associated with the 
time taken for their staff to participate. The CE of each 
participating agency was asked to nominate a senior 
member of staff to function as the liaison person (LP) 
and to be the primary point of contact for the SPIRIT 
team. An individual with extensive knowledge of broker- 
ing experience was appointed as the SPIRIT officer for 
each agency and worked with the LP to ensure that 
SPIRIT was provided effectively at that agency. 

Timing of recruitment, intervention delivery and follow-up 

As shown in figure 2, the SPIRIT programme runs for 
12 months in each health policy agency. The total meas- 
urement period is 30 months with measures being col- 
lected at the beginning of the study period, and then 
every 6 months thereafter in each agency. There are 
thus six measurement points in total. The first measure- 
ment point is a preintervention measure for all agencies 
and the final measurement point is 6 months after the 
completion of the SPIRIT programme in the last rando- 
mised pair of agencies. 

Outcomes 

Primary outcome 

1. The primary outcome is an objective measure 
(SAGE, described below) of whether the SPIRIT 
intervention results in an increase in the extent to 
which research engagement actions are undertaken 



6 



The CIPHER Investigators. BMJ Open 2014;4:e005293. doi:1 0.11 36/bmjopen-201 4-005293 



6 



Open Access 



and research is used in the development of policy 
and programme documents. 
Secondary outcomes 
1. To determine, using the self-report of staff members 
(SEER and ORACLe, described below), whether the 
SPIRIT intervention results in an increase in: 

► The capacity of health policy agencies to use 
research in terms of: (A) the value placed on the 
use of research evidence by individual policy- 
makers and by the organisation; (B) the confi- 
dence of policymakers in undertaking the 
research actions and using research; and (C) the 
systems and tools the organisation has in place; 

► Research engagement actions; 

► The use of research. 

Process evaluation: The third set of outcomes was col- 
lected as part of a process evaluation and describes the 
implementation of SPIRIT in health policy agencies 
including data around: 

A. SPIRIT components selected by each agency; 

B. Participants' engagement; 

C. The fidelity of delivery; 

D. The cost of delivery. 

Outcome measurement 

All outcome measures used in SPIRIT have been devel- 
oped specifically for the trial as we were unable to iden- 
tify any suitable measures in the literature. All measures 
were extensively piloted prior to the start of the trial and 
papers outlining their development and validation will 
be published in peer-reviewed journals. Table 2 provides 
a summary of the primary and secondary study out- 
comes and the accompanying outcome measurement 
tools. 

1. Staff Assessment of enGagement with Evidence (SAGE) is 
an objective measure of the primary outcome — the 
extent to which agencies apply research findings in 
the development of the policy and programme docu- 
ments they produce. An expert panel will be 
assembled to assign SAGE scores on the basis of 
interview data and document review. At each of the 
six data points, agencies are asked to provide the 
four policy documents finalised during the past 
6 months that best represent the use of research in 
their policy and programme development work. The 
level of measurement is the policy document. 
Information about each document is collected via a 
structured qualitative interview with an individual 
who was heavily involved in the development of the 
document, nominated by the LP. The interview runs 
for approximately 1 h. A separate interview is con- 
ducted for each nominated document. The domains 
SAGE assesses are: (A) research engagement actions 
(accessing research, appraising research (for quality 
and relevance), generating new research or analysis 
and interacting with researchers) — each of these six 
dimensions receives a score from 0 to 9. Scores can 



be summed or reported separately; (B) outcome: 
research use (four types of research use are considered: 
instrumental, tactical, conceptual and imposed). 
Each of these four dimensions receives a score from 
0 to 9. Scores can be summed or reported separately. 
2. Seeking, Engaging with and Evaluating Research (SEER) 
measures individual policymakers' capacities, 
research engagement actions and research use. SEER 
is administered via an online self-report survey to eli- 
gible policymakers identified by the LP. At each of 
the six measurement points, the LP from each 
agency provides a list of eligible policymakers to the 
research team. 
Health policy or programme staff are regarded as eli- 
gible to complete SPIRIT measures if: 

A. They write health policy documents or develop 
health programmes, or make or contribute signifi- 
cantly to policy decisions about health services, pro- 
grammes or resourcing; 

B. They are employed at a mid-level or higher in their 
agency; 

C. They are over 18 years of age and consent to partici- 
pate in the study. 

Individual agency staff members are excluded if they 
are contractors and/ or work across several of the partici- 
pating agencies. All nominated policymakers are 
emailed an invitation to participate in the SPIRIT trial 
outcome measure in question along with standard par- 
ticipant information and consent forms. SEER measures 
(A) capacity (four subscales: predisposing factors related 
to individual values, perceptions of organisational 
values, perception of organisational systems and individ- 
ual knowledge); (B) research use actions (accessing 
research, appraising, generating new research or ana- 
lyses and interacting with researchers); and (C) research 
use (conceptual, instrumental, tactical and imposed 
research use) and extent of research use. 
1. Organisational Research Access, Culture and Leadership 
(ORACLe) is used to assess each agency's capacity to 
use research findings. ORACLe data are collected via 
a structured qualitative interview with a senior staff 
member from each agency, nominated by the agency 
LP. The qualitative data are transcribed and used to 
assign scores to seven dimensions related to an orga- 
nisation's capacity to use research. Two researchers 
separately score the responses (0=no, l=yes — some or 
to a limited extent, 2=yes — ^very much so) of each 
participant to the 21 questions which make up the 
ORACLe scale. Inter-rater reliability has thus far been 
high (95%). Scores for each domain and a total 
score are then generated using an algorithm devel- 
oped via a discrete choice experiment. This experi- 
ment harnessed the views of leaders in policy and 
knowledge exchange on the most important agency- 
level factors influencing the use of evidence. 
ORACLe assesses agencies across the following seven 
domains: (A) processes that encourage or require the 
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Table 2 SPIRIT outcome measures 



Outcome 



Level of 
assessment 



Tool 



Data collection 
method 



Participants 



Domains 



Primary outcome measure 
Use of research in the Policy 
creation of policy document 
documents 



SAGE 



Interview 
(face-to-face or 
phone) plus 
collection of 
documentation 



Secondary outcome measures 

Policymakers' Individual SEER Online survey 

self-assessments policymaker 

of their research use 

capacity, actions and 

outcomes 



Organisational capacity Policy 



to use research as 
measured by the 
existing tools and 
systems to support 
research use 



agency 



ORACLe Interview 

(face-to-face or 
phone) plus 
collection of 
documentation 



One or two policymakers 
involved in the 
development of the policy 
document being 
consideredxfour 
documents at each 
measurement point 



All policymakers from 
within participating 
agencies 



One senior member of 
each policy agency, 
nominated by agency's 
leaders 



► Seeking research 
Research access efforts 
Appraisal of relevance 
Appraisal of quality 

► Using research 
Conceptual research use 
Instrumental research use 
Tactical research use 
Imposed research use 
Influence of research 

► Generating research 

Plans to generate or advocate new research 

► Influences on research use in policy 
Barriers to research use 

Enablers of research use 

► Access to, and appraisal, generation and use of research 

► Engagement with researchers 

► Value placed on research 

► Confidence in using research 

► Perceived value organisation places on use of research 

► Organisational tools and systems available to support research use 

► Policies that encourage or mandate the examination of research in 
policy and programme development 

► Tools and programmes to assist leaders of the organisation to 
actively support the use of research in policy and programme 
development 

► Strategies to provide staff with training in using evidence from 
research in policy and in maintaining these skills 

► Strategies to help staff to access existing research findings 

► Methods to generate new research evidence to inform the 
organisation's work 

► Methods to ensure adequate evaluations of the organisation's 
policies and programmes 

► Strategies to strengthen research relationships 



ORACLe, Organisational Research Access, Culture and Leadership; SAGE, Staff Assessment of enGagement with Evidence; SEER Seeking, Engaging with and Evaluating Research; SPIRIT, 
Supporting Policy In health with Research: an Intervention Trial. 
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examination of research in policy and programme devel- 
opment; (B) tools and programmes to assist leaders of 
the organisation to actively support the use of research 
in policy and programme development; (C) strategies to 
provide staff with training in using evidence from 
research in policy and in maintaining these skills; (D) 
organisational strategies to help staff to access existing 
research findings; (E) methods to generate new research 
evidence to inform the organisation's work; (F) methods 
to ensure adequate evaluations of the organisation's pol- 
icies and programmes; and (G) strategies to strengthen 
research relationships. 

Process evaluation 

A mixed methods process evaluation is conducted in 
parallel with the intervention in each of the six agencies. 
It collects data about, and analyses the interactions 
between, three domains: 

A. The delivery of the intervention, including implementa- 
tion of essential elements ( fidelity) . All intervention ses- 
sions are observed and audio recorded by process 
evaluation staff. Initial essential elements (ie, aspects 
of each intervention component that are considered 
critical to its success) are identified, and codes devel- 
oped to monitor their delivery descriptively and, 
where possible, quantitatively. Essential elements and 
codes are tested in the early stages of the programme 
delivery, refined and further tested throughout the 
trial. Revised elements are applied retrospectively to 
earlier sessions to enable comparison across interven- 
tion sites. 

B. Participation in and responses to the intervention: 
Participation attributes are recorded during session 
observations, and self-report evaluation forms are 
used to collect feedback on the relevance, applicabil- 
ity and value of each intervention session from the 
participants' point of view, including whether they 
are likely to use knowledge/skills developed in the 
session. Following the intervention, five to seven 
semi-structured interviews are conducted in each site 
which explore participants' responses to the pro- 
gramme, including changes in relation to individual 
and organisational capacity to use research, research 
engagement actions and actual research use. 

C. Contextual factors that might have affected the intervention 
and responses to it Session observations are used to 
collect data on participants' views and experiences of 
research use in their work. Preintervention interviews 
with participants explore workplace priorities and 
culture, how they perceive their organisations' 
support for research use, the role of research within 
the mix of other information and any other context- 
ual factors that affect their use of research or may 
affect how they will engage with and respond to the 
intervention, including exposure to other research 
use drivers. 

The process evaluation will complement the outcome 
measures by describing how and why the intervention 



worked or did not work, including why the same inter- 
vention strategies may have had different effects in dif- 
ferent contexts. In addition, it will aid future 
programme improvement by identifying beneficial 
design and implementation strategies and suggesting 
contextually responsive programme adaptation for dif- 
ferent recipients and settings. The full protocol for the 
SPIRIT process evaluation is described in detail 
elsewhere. 

Cost analysis 

A cost analysis is being conducted as part of SPIRIT 
because: the intervention will include non-market goods 
for which standardised cost estimates are less likely to 
exist. This cost analysis is conducted from the perspec- 
tive of the intervention provider; detailed data on the 
resources used to deliver the intervention and their 
values are collected via interviews with central interven- 
tion staff, logging of the costs involved in hiring session 
facilitators, interviews with agency liaison people regard- 
ing the time taken to complete their SPIRIT tasks, 
recording the length of interviews, monitoring the time 
taken to complete the SEER online surveys and careful 
record keeping regarding the duration of sessions. 
Microcosting (a bottom-up approach methodology) is 
being used.^"^ 

Randomisation 

The health policy agency is the unit of randomisation 
and intervention in the SPIRIT trial, with two sites ran- 
domised to start the intervention at each 'step'. The six 
participating agencies were each assigned a code. The 
list of agency codes was sent to a biostatistician who was 
not involved in any other aspect of the trial, who 
assigned each agency code a computer-generated 
random number sequence and resorted them. The first 
two agencies in the list were allocated the October 2012 
start date (step 1), the second two the April 2013 start 
date (step 2) and the final two the October 2013 start 
date (step 3). 

Blinding 

The SPIRIT staff and those investigators involved in deli- 
vering aspects of the intervention, by necessity, are not 
blind to allocation. All other investigators are blind to 
allocation, although analysis will need to know when 
sites start the intervention. They will not be informed 
which agencies are in the active intervention phase, and 
all questionnaire responses and interview records that 
they receive will be de-identified for individual partici- 
pants. The agency allocation is concealed from those 
scoring SAGE and ORACLe to limit bias; this is particu- 
larly important since the intervention team cannot be 
blinded to the active intervention agencies. 

Statistical methods 

Analysis will be undertaken using generalised linear 
models with a link function and error distribution 
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appropriate to the outcome measure. In each model, 
the intervention effect will be estimated as the differ- 
ence between the postintervention and preintervention 
levels of the outcome after adjusting for time. The unit 
of analysis will be the agency for the SAGE and ORACLe 
outcome measures and the individual for the SEER 
outcome measure. Since there are too few clusters to 
employ either a linear mixed model or a generalised 
estimating equation (GEE) approach, all models will 
adjust for the clustering by using agency as a fixed 
effect. 

For the study design outlined in figure 2 and with the 
assumption of an intraclass correlation coefficient of 
0.01, a total sample size of 144 documents, 4 per site per 
time period, will give 89% power to detect a 1 SD differ- 
ence in the mean SAGE total score at the 5% signifi- 
cance level. 

Data management and monitoring 

Data for the SEER measure are entered online by parti- 
cipants. SEER is primarily multiple choice. In order to 
enhance data quality, the online survey has built in skip 
functions and participants are unable to progress from 
one page to the next until all questions are answered. 
Data from SAGE and ORACLe are transcribed (the 
scoring and checking procedure was described in the 
outcome measures section). 

All data and documents are stored on password pro- 
tected computers and on a network server accessible 
only to the researchers. Any paper documents are stored 
in a securely locked filing cabinet located at the Sax 
Institute, which has day-to-day management of CIPHER. 
The Sax Institute is a secure facility requiring a unique 
staff access code to enter; visitors are escorted on-site 
and are required to sign in. 

A data monitoring committee was not required for 
this trial as the risk of harm to participants is negligible. 

Ethics and dissemination 

All participants provide electronic consent to participate 
in outcome measures via return email, and signed 
consent for process evaluation data collection. All par- 
ticipating agencies and individuals are free to decline to 
participate in any and all aspects of SPIRIT at any time 
with no explanation required (example participant 
information and consent forms in online supplementary 
appendix 1). 

Participating agencies receive feedback on their results 
at regular intervals during the active phase of the inter- 
vention and again at the conclusion of the trial. The 
findings of this study will be disseminated more broadly 
through peer-reviewed publications (authorship deter- 
mined by BMJ guidelines) and presentations at confer- 
ences and used to inform future strategies. 
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